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Abstract 


In  the  Appraiser  Program  of  the  Software  Engineering  Institute  (SEI),  authorized  Lead  Asses¬ 
sors  lead  Capability  Maturity  Model-Based  Appraisals  for  Internal  Process  Improvement 
(CBA IPI).  At  the  conclusion  of  each  assessment,  they  are  required  to  submit  certain  artifacts 
to  the  SEI.  Data  from  assessments  is  recorded  to  provide  the  community  with  information  on 
the  state  of  the  software  community’s  process  maturity,  as  related  to  the  Capability  Maturity 
Model  (CMM)  for  Software  Version  1.1.  These  data  can  be  viewed  on  the  SEI  Web  site: 
<URL  http://  www.sei.cmu.edu/sema/profile.html>. 

Additional  feedback  data  are  required  of  a  Lead  Assessor  in  order  to  monitor  the  consistency 
of  use  of  the  assessment  method  for  quality  control  purposes.  Data  are  collected  from  Lead 
Assessors,  assessment  team  members,  and  sponsors  of  the  assessments.  The  results  reported 
in  this  document  reflect  information  sent  to  the  SEI  by  Lead  Assessors  through  a  Lead  Asses¬ 
sor’s  Requirements  Checklist.  The  checklist  aids  the  Lead  Assessors  in  keeping  track  of  their 
implementation  of  each  of  the  method’s  requirements.  The  checklist  also  provides  informa¬ 
tion  back  to  the  community  regarding  metrics  being  reported  by  Lead  Assessors;  this  helps  in 
more  effective  planning  for  future  assessments.  In  addition,  the  checklist  acts  as  a  quality 
control  mechanism  to  monitor  the  consistency  of  use  of  each  of  the  method’s  activities. 

Thanks  to  the  Lead  Assessors  who  contributed  the  data  in  order  that  it  can  be  shared  with 
other  Lead  Assessors  and  the  community. 
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1  Purpose  of  Document 


The  main  purpose  of  this  document  is  to  consolidate  and  analyze  information  from  Lead  As¬ 
sessor  Requirements  Checklists  that  have  been  submitted  by  Lead  Assessors  in  assessments 
conducted  using  the  Capability  Maturity  Model®-Based  Appraisal  for  Internal  Process  Im¬ 
provement  (CBA IPI)  method. 

The  audience  for  this  document  is  the  community  of  software  developers  who  are  contem¬ 
plating  having  a  CBA  IPI  assessment  in  their  organization  and  Lead  Assessors  who  are  inter¬ 
ested  in  learning  more  about  others’ experiences  in  order  to  improve  their  own  planning  and 
use  of  the  CBA  EPI  method. 

1.1  Data  Analysis  Process 

A  total  of  83  Lead  Assessor  Requirements  Checklists  were  completed  and  submitted  as  of 
November  1,  1999,  for  assessments  conducted  between  July  1998  and  October  1999.  Al¬ 
though  there  were  over  300  CBA  IPI  reports  returned  to  the  SEI  for  this  time  period,  the  Lead 
Assessor  Requirements  Checklist  was  a  new  requirement  added  in  late  1998.  Monitoring  of 
the  return  of  this  checklist  along  with  other  required  feedback  forms  is  being  enforced  con¬ 
sistently  at  this  time. 

1.1.1  Data  Entry 

A  Microsoft  Access  database  was  designed,  and  the  data  was  entered  manually.  The  SEI 
Lead  Assessor  Web  Center  is  under  development  and  will  be  available  in  the  second  quarter 
of  2000,  so  that  the  data  can  be  entered  directly  online  by  the  Lead  Assessors.  This  MS  Ac¬ 
cess  database  was  designed  so  that  existing  data  could  be  subsequently  imported  into  the 
Lead  Assessor  Web  Center  database. 

1.1.2  Data  Analysis 

The  data  was  checked  for  consistency  and  corrected  accordingly.  Details  regarding  which 
data  fields  have  been  modified  are  provided  where  appropriate  throughout  this  document. 

The  data  are  analyzed  using  Microsoft  Excel  pivot  tables  and  charts,  and  manual  counting 
where  necessary.  For  various  numerical  data  elements,  where  the  range  of  values  is  very 


p  Capability  Maturity  Model  is  registered  in  the  U.S.  Patent  and  Trademark  Office. 
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large,  the  values  are  grouped  together  to  create  ranges  for  easier  analysis  and  visualization. 
For  free  text  data,  the  information  provided  is  reviewed  to  identify  a  few  major  categories. 

1.1.3  Data  Reporting 

In  this  report,  the  results  of  analysis  are  typically  presented  with  a  histogram  or  pie  chart,  de¬ 
pending  on  which  provides  the  most  comprehensive  view  of  the  data  involved.  In  many 
cases,  the  minimum  and  maximum  value  as  well  as  the  mode  (most  frequently  occurring) 
value  is  highlighted.  Where  appropriate,  the  average  and  median  values  are  also  computed. 

1 .2  Document  Overview 

This  document  is  organized  based  on  the  format  of  the  hardcopy  Lead  Assessor  Requirements 
Checklist.  There  are  four  major  sections  in  the  checklist: 

•  planning  the  assessment 

•  conducting  the  assessment 

•  reporting  results 

•  additional  questions 


The  findings  for  each  of  these  major  sections  are  presented  in  the  chapters  following.  In  each 
chapter,  an  analysis  of  the  results  for  each  question  that  is  significant  or  meaningful  is  pre¬ 
sented.  The  question  as  shown  in  the  checklist  is  first  presented,  followed  by  a  graph  that 
provides  a  visual  indicator,  and  text  that  describes  the  analysis  process  and  results. 


10 
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2  Planning  the  Assessment 


2.1  Assessment  Materials 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Material  for  each  assessment  must  be  pur¬ 
chased  from  the  SEI. 

Material  for  this  assessment  was  obtained  via: 

•  Single  kit 

•  Quantity  kit 

Figure  1:  How  Assessment  Materials  Were  Obtained 

Out  of  the  83  forms  submitted,  53  indicated  that  a  quantity  kit  was  used,  while  29  indicated 
that  a  single  kit  was  purchased.  One  entry  did  not  specify  how  the  assessment  materials  were 
obtained. 
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2.2  Team  Composition 
2.2.1  Team  Size 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

The  team  shall  have  4  to  10  team  members.  At 
least  one  member  must  be  from  the  organiza¬ 
tion  being  assessed. 

•  Total  number  of  team  members 

•  Number  of  team  members  from  the  as¬ 
sessed  organization 
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Figure  2:  Total  Number  Of  Team  Members 

The  histogram  above  indicates  that  the  team  size  ranges  from  4  to  13  members.  Three  out  of 
83  assessments  (~4%)  exceeded  the  team  size  of  4  to  10  members  recommended  by  the  SEI. 
The  average  team  size  is  seven,  which  is  also  the  most  frequently  occurring  team  size  (mode) 
as  well  as  the  median.  The  histogram  above  is  skewed  right  indicating  that  there  are  fewer 
large  teams  (>  9  members). 
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2.2.2  Proportion  of  Members  from  Assessed  Organization 


Percentage  of  team  members  from  assessed  organization 


Figure  3:  Proportion  of  Members  from  Assessed  Organizations 


The  CBA IPI  method  requires  that  at  least  one  member  of  the  team  must  be  from  the  assessed 
organization.  This  means  a  range  of  10-25%  of  a  team  of  4  to  10  members.  The  distribution 
above  indicates  that  all  83  teams  met  this  requirement.  There  are  some  teams  comprised  of  all 
members  from  the  assessed  organization. 

2.2.3  Team  Member  Selection  Guidelines 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Team  members  must  meet  the  selection  guide¬ 
lines. 

Upon  checking  credentials  of  assessment  team 
members,  how  would  you  rate  the  team’s  expe¬ 
rience  level  against  the  recommended  guide¬ 
lines? 

(rate  on  a  scale  from  1  to  5: 

1-do  not  meet  guidelines;  5-exceed  the  guide¬ 
lines) 

CMU/SEI-2000-TR-005 
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Rating  of  team  members  based  on  selection 
guidelines 


Figure  4:  Rating  of  Team  Members  Based  on  Selection  Guidelines 

From  Figure  4  above,  it  is  clear  that  most  teams  do  not  have  problems  with  finding  team 
members  who  meet  the  selection  guidelines.  About  half  the  teams  are  comprised  of  members 
who  are  above  average,  based  on  the  guidelines  (rating  4)  and  no  teams  have  difficulty 
meeting  the  selection  guidelines  (rating  1  or  2). 


2.3  Assessed  Organization 

2.3.1  Business  Goals 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

The  assessment  is  discussed  with  the  sponsor  to 
understand  the  business  goals. 

The  business  goals  of  the  sponsor  were  deter¬ 
mined  to  be:  (please  describe) 

After  reviewing  the  various  business  goals  of  the  83  assessments  conducted,  10  most  fre¬ 
quently  occurring  goals  are  identified,  and  all  the  business  goals  are  classified  under  each  of 
these  goals.  The  following  table  shows  the  10  most  frequently  occurring  goals,  and  the  num¬ 
ber  of  organizations  that  had  a  business  goal  somewhat  related.  Note  that  each  assessed  or¬ 
ganization  may  have  specified  more  than  one  business  goal. 

From  Figure  5,  it  is  clear  that  improving  quality  and  productivity  (“Faster,  better,  cheaper”)  is 
the  most  frequently  stated  goal.  Identifying  improvement  areas  and  attaining  Level  2  Matur¬ 
ity  are  also  frequent  business  goals. 
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General  goal 

Number 

“Faster,  better,  cheaper.” 

This  goal  emphasizes  improving  quality,  productivity  and  customer  satisfaction.  It  is 
probably  the  overall  goal  for  software  process  improvement;  the  following,  more 
specific  goals  are  generally  based  upon  it. 

20 

Verify  improvement  results. 

This  goal  implies  that  some  work  has  been  done  for  software  process  improvement, 
and  an  assessment  is  done  to  measure  or  verify  its  success.  This  includes  goals  to 
measure  process  improvement  results  and  acknowledge  improvements  achieved. 

12 

Identify  improvement  areas  /  opportunities. 

This  goal  focuses  on  identifying  the  strengths  and  weaknesses  of  the  organization’s 
software  process. 

17 

Establish  baseline  for  process  improvement 

The  emphasis  of  this  goal  is  to  establish  a  baseline  to  start  improvement  work  so  as  to 
guide  tracking  of  software  process  improvement.  The  baseline  may  be  related  to  any 
Capability  Maturity  Model  (CMM®)  maturity  level. 

14 

Attain  Level  2  Maturity 

17 

Attain  Level  3  Maturity 

7 

Attain  Level  4  or  5  Maturity 

2 

Generate  management  and  staff  support  for  software  process  improvement 

2 

Meet  contractual  requirement 

2 

Unknown 

12 

The  business  goal  is  not  stated  or  reference  is  made  to  another  document. 

E  CMM  is  registered  with  the  U.S.  Patent  and  Trademark  Office. 
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Business  goals 


</> 

C 


Figure  5:  Business  Goals 


2.3.2  Organization  Size 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

The  organization  scope  including  selected  proj¬ 
ects  and  participants  must  be  determined. 

There  are  persons  in  this  organization 

with  technical  and  managerial  responsibilities 
for  software  development. 

The  organization  scope  is  determined  to  be: 

16 
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Number  of  software  development  people  in  the 
organization 


Figure  6:  Number  of  People  in  the  Organization 

The  number  of  people  in  the  assessed  organization  with  technical  and  managerial  responsi¬ 
bilities  for  software  development  range  from  a  minimum  of  5  people  to  a  maximum  of  934 
people  in  the  83  assessments  performed.  The  above  figure  shows  the  number  of  organizations 
that  fall  into  each  range  of  organization  size.  More  than  half  the  assessed  organizations  have 
150-or-fewer  people  who  have  technical  and  managerial  responsibilities  for  software  devel¬ 
opment. 

2.3.3  CMM  Scope 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

The  CMM  scope  (KPAs  to  be  examined) 
must  be  determined. 

The  CMM  scope  (KPAs  to  be  examined)  is  de¬ 
termined  to  be: 

Level  2:  RM,  SPP,  SPTO,  SSM,  SQA,  SCM 

Level  3:  OPF,  OPD,  TP,  ISM,  SPE,  IC,  PR 

Level  4:  QPM,  SQM 

Level  5:  DP,  TCM,  PCM 

Responses  to  this  question  in  the  Lead  Assessor  Requirements  Checklist  may  be  classified 
into  the  following  categories: 

•  Level  2  only  (without  SSM) 

•  Level  2  only  (with  SSM) 

•  Level  2  (without  SSM)  +  Level  3 

•  Level  2  (with  SSM)  +  Level  3 

This  includes  those  organizations  that  selected  all  of  the  KPAs  in  Level  2  including  Soft¬ 
ware  Subcontract  Management  (SSM),  and  some  or  all  of  the  KPAs  from  Level  3.  No 
KPAs  from  Levels  4  or  5  were  selected. 
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•  Level  2  +  Level  3  +  Level  4 

This  includes  those  that  selected  all  of  the  KPAs  in  Level  2  (with  or  without  SSM),  some 
or  all  the  KPAs  from  Level  3,  and  all  KPAs  from  Level  4.  No  KPAs  from  Level  5  were 
selected. 

•  Level  2  +  Level  3  +  Level  4  +  Level  5 

This  includes  those  that  selected  all  of  the  KPAs  in  Level  2  (with  or  without  SSM),  some 
or  all  the  KPAs  from  Level  3,  and  all  KPAs  from  Level  4  and  Level  5. 


Figure  7  below  shows  the  number  of  assessed  organizations  with  CMM  scope  in  each  of  the 
above  categories.  Almost  40%  of  the  assessed  organizations  selected  all  KPAs  from  Level  2 
(excluding  SSM)  and  some  or  all  KPAs  from  Level  3  as  their  CMM  scope.  It  is  also  noted 
that  among  the  organizations  whose  CMM  scope  included  Level  2  and/or  Level  3  only,  about 
one-third  determine  Software  Subcontract  Management  (SSM)  to  be  not  applicable. 


CMM  Scope 


only  only  (without  (with  Level  3  +  Level  3  + 

(without  (with  SSM)  +  SSM)  +  Level  4  Level  4  + 
SSM)  SSM)  Level  3  Level  3  Level  5 


Figure  7:  CMM  Scope 

Level  2  KPAs  were  typically  all  selected  with  or  without  SSM.  Level  4  and  5  KPAs  were 
typically  all  selected  when  the  level  was  included  in  the  CMM  scope.  Level  3  has  the  greatest 
variation  in  terms  of  the  KPAs  selected.  Figure  8  provides  a  breakdown  of  the  KPAs  that  are 
selected  in  Level  3.  The  last  column  reflects  the  number  of  organizations  with  a  CMM  scope 
that  includes  all  of  the  KPAs  in  Level  3.  It  is  not  to  be  compared  with  the  other  columns. 


The  histogram  indicates  that  Integrated  Software  Management  (ISM)  is  the  least  frequently 
selected  KPA  in  Level  3,  followed  by  Intergroup  Coordination  (IC)  and  Software  Product 
Engineering  (SPE). 
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Level  3  KPAs  selected  in  CMM  scope 


KPAs 


Figure  8:  Level  3  KPAs  selected  in  CMM  Scope 


2.4  Training 

2.4.1  CMM  Training 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

All  team  members  must  receive  the  SEI  In¬ 
troduction  to  CMM  course  or  equivalent 

Was  the  SEI  Introduction  to  CMM  course  used  in 

its  entirety? _ yes  _ no 

If  not,  what  CMM  course  was  used  (e.g.,  source, 
instructor,  date) 

For  this  question  in  the  Lead  Assessor  Requirements  Checklist,  there  seems  to  be  some  con¬ 
fusion  in  the  responses.  Some  responded  with  a  “yes”  to  the  question,  and  yet  provided  a 
course  name  that  is  not  the  SEI  Introduction  to  CMM  course.  Hence,  the  data  was  reviewed 
and  corrected  for  consistency.  A  new  data  value,  “some”,  was  also  introduced  in  addition  to 
the  “yes”  or  “no”  answer  to  accommodate  entries  where  both  “yes”  and  “no”  were  indicated 
due  to  some  team  members  who  had  SEI  Introduction  to  CMM  training,  but  others  who  had 
other  training. 
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CMM  training  used 


Figure  9:  CMM  Training  Used 

The  pie  chart  above  shows  the  distribution  of  teams  that  had  the  SEI  Introduction  to  CMM 
training,  those  who  had  other  training,  and  those  that  had  a  combination  of  both.  For  the 
teams  with  non-SEI  training,  the  CMM  training  provided  is  typically  in-house  developed 
courses;  otherwise,  it  may  be  third-party  courses,  or  courses  taught  by  the  Lead  Assessor. 


2.4.2  CBA IPI  Team  Training 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

All  team  members  must  receive  the  SEI  CBA 
IPI  Team  Training  course. 

Was  the  SEI  CBA  IPI  Team  Training  used  in  its 
entirety? _ yes  _ no.  If  no,  describe  modifi¬ 

cations: 

Dates  and  time  (number  of  hours)  allocated  for 
Team  Training: 

Planned: 

Actual: 

Did  you  supplement  the  Team  Training  with  your 

own  material? _ yes _ no.  If  yes,  what  areas 

did  you  supplement? 

How  many  team  members  participated  in  CBA 

IPI  Team  Training?  If  anv  team  members 

were  waived  from  team  training,  please  indicate 
the  Lead  Assessor  and  the  dates  that  this  team 
member  received  prior  CBA  IPI  Team  Training: 
Team  members  waived  (names,  prior  training, 
instructor  &  dates): 

20 
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2.4. 2.1  Time  Spent  in  CBA  IPI  Team  Training 


Delivery  time  for  CBA  IPI  team  training 
(range  of  hours) 
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Figure  10:  Delivery  Time  for  CPA  I  PA  Team  Training 


The  number  of  hours  spent  on  CBA  IPI  team  training  ranged  from  3  hours  to  40  hours.  The 
various  delivery  times  are  grouped  into  ranges  shown  in  Figure  10.  Most  teams  fall  under  the 
16-20  hours  range.  The  complete  data  indicates  that  21  teams  have  team  training  consuming 
20  hours,  and  11  teams  have  training  that  consumes  16  hours. 
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2.4.2. 2  Team  Members  Waived  from  Team  Training 


Number  of  team  members  waived  from  CBA IPI 
team  training 


Figure  11:  Number  of  Team  Members  Waived  from  CBA  IPI  Team  Training 


The  number  of  team  members  waived  from  CBA  IPI  team  training  was  obtained  by  sub¬ 
tracting  the  number  who  participated  in  team  training  from  the  team  size.  In  four  of  the  cases 
this  produced  a  negative  number;  these  involved  people  who  were  not  part  of  the  team  but 
who  participated  in  the  team  training.  For  such  cases,  it  is  assumed  that  the  number  of  people 
who  participated  in  the  team  training  included  all  the  team  members  and  the  number  waived 
is  zero.  The  histogram  above  indicates  that  most  teams  (55%)  do  not  have  any  team  members 
waived  from  CBA  IPI  team  training.  About  20%  of  the  teams  have  one  person  waived  and 
25%  of  the  teams  have  two  or  more  persons  waived. 
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2.4.2.3  Supplementary  Materials 


I 


Supplementary  materials  provided  for 
CBA IPI  team  training 

Unknown 


Figure  12:  Supplementary  Materials  Provided  for  CBA  IPI  Team  Training 

Of  the  83  assessments  reported,  22  Lead  Assessors  did  not  use  any  supplementary  materials 
for  team  training,  while  58  Lead  Assessors  did. 

Classifying  the  types  of  supplementary  materials  used  will  be  useful  in  identifying  the  areas 
of  improvement  for  the  CBA  IPI  Team  Training.  Out  of  those  who  did  provide  supplementary 
materials,  the  types  of  supplementary  materials  provided  can  be  classified  as  follows: 

1 .  planning  and  team  building 

2.  automation  tools  (including  electronic  worksheets) 

3.  KPA  worksheets  and  other  templates 

4.  document  review  techniques/exercises 

5.  interview-related  techniques/exercises 

6.  consolidation  techniques/exercises 

7.  exercises,  examples  and  case  studies  (usually,  related  to  the  assessed  organizations) 

8.  rating/scoring  system 

9.  materials  with  respect  to  small  organizations 

10.  unknown  (either  unspecified  or  too  general  to  be  categorized) 


The  distribution  of  the  types  of  supplementary  materials  used  is  shown  in  Figure  13  below. 
Automation  tools  and  exercises  are  the  most  frequent  types  of  supplements  to  the  team  train¬ 
ing.  This  is  to  be  expected,  since  various  organizations  have  their  own  ways  of  automating 
processes,  and  it  is  useful  to  tailor  exercises  to  the  context  and  data  of  the  assessed  organiza¬ 
tion.  The  next  most  frequent  types  of  supplementary  materials  used  are  in  the  areas  of  plan¬ 
ning  and  team  building,  consolidation  and  interview  techniques/exercises. 
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Number  of  assessments 


Types  of  Supplementary  materials 


i 


Figure  13:  Types  of  Supplementary  Materials 
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3  Conducting  the  Assessment 


3.1  Maturity  Questionnaires 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Administer  maturity  questionnaires  for  at 
least  the  project  leaders  from  the  selected 
projects. 

How  many  questionnaires  were  administered? 

Number  of  questionnaires  administered 
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Figure  14:  Number  of  Questionnaires  Administered 


The  histogram  above  shows  the  range  in  the  number  of  maturity  questionnaires  that  were 
administered  in  the  83  assessments.  For  most  assessments,  one  to  five  questionnaires  are  ad¬ 
ministered.  Only  one  reported  not  using  the  questionnaire.  The  actual  numbers  range  from  0 
to  47  questionnaires. 

According  to  the  CBA  EPI  requirement,  maturity  questionnaires  should  be  administered  to  at 
least  the  project  leaders  from  the  selected  projects. 
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3.2  Interviews 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Conduct  interviews: 

-  project  leaders  (individual  interviews) 

-  middle  managers  (group  interviews) 

-  functional  area  representatives  (group  inter¬ 
views) 

Indicate  number  and  duration  of  each  type  of  in¬ 
terview: 

-  project  leaders  : 

-  middle  managers  : 

-  functional  area  representatives  : 

-  total  number  of  interviewees: 

Total  number  of  interviewees 


Figure  15:  Total  Number  of  Interviewees 


3.3  Documents  Reviewed 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Conduct  document  review  at  a  minimum  for 
each  KPA  goal  within  the  assessment  scope. 

Approximately  how  many  documents  have  been 
reviewed? 

Documentation  must  be  examined  at  least  for 
each  goal  for  each  KPA  within  the  assess¬ 
ment 

Documentation  was  examined  for: 
each  key  practice 
each  goal 

The  two  questions  above  are  not  found  sequentially  in  the  Lead  Assessor  Requirements 
Checklist.  However,  it’s  been  noted  that  they  seem  repetitive,  and  they  will  be  combined 
when  the  checklist  is  put  on  the  Web  Center.  Most  responses  provided  the  same  value  for  the 
total  number  of  documents  examined  in  these  two  questions. 
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The  number  of  documents  reviewed  ranged  from  10  to  1500  in  the  83  assessments  analyzed. 
There  were  13  feedback  forms  that  did  not  indicate  the  number  of  documents  reviewed.  In 
some  cases  the  documents  were  not  counted;  in  others,  documents  were  reviewed  on-line  and 
hence,  were  not  physically  counted.  Most  assessment  teams  reviewed  10  to  100  documents. 

The  data  obtained  from  the  above  question  may  not  be  precise,  as  there  are  different  ways  of 
counting  the  number  of  documents.  For  example,  some  forms  reflect  the  number  of  binders 
reviewed,  while  others  provide  only  an  estimate  of  how  many  physical  documents  have  been 
reviewed.  Clearer  definitions  are  needed  in  order  to  get  a  consistent  count  of  documents  re¬ 
viewed. 


Figure  16:  Number  of  Documents  Reviewed 


Although  the  CBA IPI  method  only  requires  documentation  to  be  examined  for  each  goal 
within  the  assessment  scope,  at  a  minimum,  most  assessments  (77  out  of  83,  or  93%)  go  be¬ 
yond  that  to  the  key  practice  level. 
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Level  of  Documentation  Examined 


Figure  17:  Level  of  Documentation  Examined 


3.4  Level  of  Data  Collection 

CBA  IPI  Requirement  I  How  This  Assessment  Was  Implemented 

Collect  data  for  each  key  practice  for  each  _ Data  was  collected  only  at  the  goal  level. 

KPA  within  the  assessment  scope.  _ Data  was  collected  for  each  key  practice. 

_ Data  was  collected  for  each  subpractice. 


Figure  18:  Level  of  Data  Collection 
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Data  collection  is  required  at  the  key  practice  level  in  CBA  IPI.  However,  there  has  been  an¬ 
ecdotal  evidence  of  more  stringent  data  collection  being  required  at  the  subpractice  level. 
This  is  a  potential  problem  due  to  the  prescriptive  nature  of  the  subpractices  for  which  the 
CMM  is  not  intended.  This  does  not  appear  to  be  a  pervasive  problem  since  only  four  as¬ 
sessments  indicate  that  data  is  collected  at  the  subpractice  level. 

The  histogram  above  shows  that  98%  of  the  83  assessments  conducted  met  the  CBA  IPI  re¬ 
quirement  for  data  collection  for  each  key  practice  for  each  KPA.  In  only  one  assessment  was 
data  collected  only  at  the  goal  level. 

3.5  Data  Corroboration 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Data  was  corroborated  coming  from  at  least 
two,  independent  sources  at  different  ses¬ 
sions. 

The  entire  assessment  team  determined  that  each 
observation  was  valid  (accurate,  corroborated, 
consistent). 

_ yes _ no.  If  not,  please  explain: 

For  this  question,  the  response  was  unanimously  “yes”  for  all  83  assessments. 


3.6  Observations 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Each  key  practice  for  each  KPA  within  the 
assessment  scope  must  be  determined  to  be 
sufficiently  covered  with  observations 
crafted  from  data  collected. 

_ number  of  observations  were  created  (total). 

The  assessment  team  determined  sufficient  cov¬ 
erage  for  each  key  practice  for  each  KPA  within 

the  assessment  scope. _ yes _ no.  If  not, 

please  explain. 
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Number  of  Observations 
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Figure  19:  Number  of  Observations 


Sufficiency  of  coverage 

Not  sufficient 


6% 


Figure  20:  Sufficiency  of  Coverage 

The  total  number  of  observations  created  in  the  83  assessments  examined  ranged  from  7  to 
3000.  The  histogram  above  shows  the  distribution  in  ranges.  Most  assessments  created  200 
observations  or  less,  and  the  average  number  of  observations  is  approximately  300. 

With  respect  to  the  issue  of  sufficient  coverage  for  each  key  practice  for  each  KPA  within  the 
assessment  scope,  the  pie  chart  above  indicates  only  6%  (5  out  of  83  assessments)  had  prob¬ 
lems  getting  sufficient  coverage.  Out  of  these  five  assessments,  two  did  not  expect  to  get  full 
coverage  for  the  higher  levels  of  maturity  that  were  included  in  the  assessment  scope.  Two 
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did  not  have  enough  data  to  cover  one  specific  KPA.  One  assessment  determined  a  specific 
goal  of  a  KPA  to  be  not  applicable  and  did  not  cover  it.  However,  individual  goals  are  not 
subject  to  be  tailored  out  for  non-applicability  unless  the  entire  KPA  has  previously  been  de¬ 
termined  to  be  not  applicable. 


3.7  Draft  Findings 


CBA  IPI  Requirement 


How  This  Assessment  Was  Implemented 


Conduct  draft  finding  presentati 


number  of  draft  findings  were  presented  at 
(how  many)  draft  finding  presentations 
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Figure  22:  Number  of  Draft  Finding  Presentations 

The  number  of  draft  findings  ranges  from  1  to  236.  However,  the  two  figures  above  indicate 
that  50%  of  the  assessments  have  60  or  fewer  draft  findings,  and  that  there  are  typically  two 
draft-finding  presentations. 


3.8  Type  of  Ratings 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Ratings  must  be  made  based  on  suffi¬ 
ciently  covered  key  practices  mapped 
to  the  KPA  goals,  (maturity  level  rating 
is  optional) 

Ratings  were  done  by  the  assessment  team  for: 

_ maturity  level 

_ all  KPAs  within  the  scope 

-  except  ("KPAs  not  rated): 

_ each  goal  for  each  of  above  KPAs 

_ each  key  practice  within  each  of  above  KPAs 

(tailoring  option) 

Figure  23  below  presents  the  responses  for  this  question.  About  50%  of  the  assessments  had 
ratings  done  for  the  maturity  level,  all  KPAs  within  the  scope,  and  each  goal  for  each  KPA. 
37%  of  the  assessments  had  ratings  done  for  all  the  levels  (each  key  practice  within  each 
KPA  in  addition  to  the  maturity  level,  KPA  and  goals). 
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Level  of  Ratings  Done 


+goal 


Figure  23:  Level  of  Ratings  Done 


3.9  Decision  Making  Strategy 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

Consensus  is  the  decision-making 
strategy  of  an  assessment  team. 

Decisions  were  made  by  consensus  of  the  assessment 
team. _ yes _ no.  If  not,  please  explain. 

The  data  from  the  83  assessments  indicate  that  all  the  teams  used  consensus  as  their  decision¬ 
making  strategy. 
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4  Reporting  Results 


4.1  Sponsor  Participation 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

A  final  findings  briefing  must  be  given 
to  the  sponsor. 

The  sponsor  attended  the: 

_ Opening  Meeting 

_ Final  Findings  Briefing 

_ Executive  Session 

Sponsor  Participation 


□  All 

■  Opening  meeting  +  Final 
findings 

□  Final  findings  +  Executive 
session 

□  Final  findings  only 

■  Opening  meeting  only 


Figure  24:  Sponsor  Participation 


In  75%  of  the  assessed  organizations,  the  sponsor  attended  all  three  of  the  meetings.  Only  2% 
(2  out  of  83)  of  the  assessments  had  sponsors  who  only  attended  the  Opening  Meeting  and 
did  not  meet  the  CBA  IPI  requirement. 
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4.2  Reports  Submitted 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

The  final  findings  briefing  along 
with  the  KPA  profile  must  be  sub¬ 
mitted  to  the  SEI  within  30  days  of 
the  conclusion  of  the  assessment. 

The  following  are  being  submitted  to  the  SEI: 

_ PAIS  report  with  Organization  and  Project 

Questionnaires 

_ Final  findings  briefing  with  KPA  profile 

_ Required  feedback  forms  (inch  this  checklist) 

_ Assessment  plan 

Figure  25:  Reports  Submitted  to  the  SEI 

The  CBA IPI  requirement  states  that  the  final  findings  briefing  along  with  the  KPA  profile 
must  be  submitted  to  the  SEI.  The  responses  from  the  83  assessments  indicated  that  this  was 
done  for  all  of  the  83  the  assessments.  77  out  of  the  83  assessments  submitted  all  of  the  other 
documents  as  well,  while  the  rest  either  did  not  submit  the  assessment  plan  or  failed  to  sub¬ 
mit  both  the  assessment  plan  and  the  feedback  forms. 
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5  Additional  Questions 


5.1  Length  of  Assessments 

In  the  beginning  of  the  Lead  Assessor  Requirements  Checklist,  the  beginning  and  end  dates 
of  the  CBA  IPI  are  recorded.  Based  on  these  dates,  the  number  of  days  for  each  assessment 
can  be  computed,  ignoring  any  weekends  or  public  holidays.  The  following  histogram  shows 
the  distribution  of  the  number  of  assessment  days  for  the  83  assessments. 
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Figure  26:  Number  of  Assessment  Days 

The  figure  above  indicates  that  the  most  frequent  assessment  length  is  5  days  (20  assess¬ 
ments.  approximately  24%).  This  is  followed  by  assessments  that  last  from  8  to  10  days.  The 
average  number  of  days  for  an  assessment  is  nine  days  (based  on  a  total  of  82  assessments, 
excluding  the  one  that  did  not  provide  the  assessment  dates). 


5.2  Team  Hours 


CBA  IPI  Requirement 

How  This  Assessment  Was  Implemented 

How  many  team-hours  (total  number  of 
hours  the  team  worked  together)  were  spent 
in  pre-onsite  activities — e.g.,  team  training, 
document  review,  scripting  questions? 

How  many  team-hours  were  spent  in  on-site 
activities  e.g.,  interviews,  data  consolidation, 
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findings  preparation  and  presentation? 

How  many  team-hours  were  spent  in  data 
consolidation  activities? 

The  responses  for  these  questions  in  the  Lead  Assessor  Requirements  Checklist  had  a  very 
wide  variance. 

These  figures  were  adjusted  for  consistency.  There  were  different  interpretations  of  the  term 
“team-hours.”  Some  Lead  Assessors  interpreted  “team-hours”  to  mean  total  person-hours, 
and  multiplied  the  number  of  team  members  by  the  time  spent  by  each  team  member.  The 
intention  of  these  questions  is  that  “team-hours”  refer  to  the  total  amount  of  time  that  the 
team  spends  together  on  team  activities  in  that  particular  phase  (pre-onsite,  on-site  or  report¬ 
ing).  The  questions  will  be  clarified  to  eliminate  confusion  in  the  future. 

The  first  two  questions  are  intended  to  identify  the  distribution  of  team-hours  spent  in  pre- 
onsite  versus  on-site  activities  of  a  CBA IPI  assessment.  The  third  question,  however,  refers 
specifically  to  consolidation  activities  only,  which  is  a  part  of  the  on-site  activities. 


Minimum 

team-hours 

Median  team- 
hours 

Maximum 

team-hours 

Pre-onsite  activities 

5 

35 

70 

On-site  activities 

32 

62 

117 

Consolidation  activities  (in¬ 
cluded  in  on-site  activities) 

3 

19 

47 

Total  Team  Hours 

48 

97 

198 

5.2.1  Pre-onsite  and  On-Site  Time  Distribution 

Approximately  50%  of  the  teams  spend  less  than  35  hours  on  pre-onsite  activities  and  less 
than  62  hours  on  on-site  activities,  shown  as  the  medians  on  the  table  above.  The  proportion 
of  pre-onsite  activities  to  on-site  activities  is  indicated  from  data  on  80  assessments,  discard¬ 
ing  the  3  assessments  that  did  not  have  responses  to  all  3  questions  above.  On  the  average, 
teams  spend  34%  of  the  assessment  time  on  pre-onsite  activities  and  66%  on  on-site  activi¬ 
ties. 
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Figure  27:  Average  Distribution  of  Total  Team  Hours 


5.2.2  Team  Hours  Spent  on  Consolidation  Activities 

It  has  been  noted  that  consolidation  activities  are  a  major  time  requirement  during  the  on-site 
phase.  Therefore,  a  question  is  included  in  the  Lead  Assessor  Requirements  Checklist,  which 
explicitly  asks  about  time  spent  on  consolidation  activities.  This  information  makes  it  possi¬ 
ble  to  study  the  proportion  of  time  required  for  the  on-site  activities  relative  to  consolidation. 


Figure  28:  Team  Hours  Spent  on  Consolidation  Activities 

Figure  28  shows  the  range  of  team  hours  spent  by  various  teams.  32  teams  spent  less  than  20 
hours  on  consolidation  activities,  while  24  teams  spent  between  21  to  40  hours. 
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It  is  also  interesting  to  consider  the  team  hours  spent  on  consolidation  activities  with  respect 
to  the  total  team  hours.  This  percentage  may  be  calculated,  assuming  that:  there  is  no  overlap 
in  the  values  provided  for  the  pre-onsite  and  on-site  activities;  and  the  team  hours  provided 
for  the  consolidation  activities  is  a  subset  of  that  for  the  on-site  activities.  The  sum  of  the  val¬ 
ues  provided  for  the  pre-onsite  activities  and  on-site  activities  is  used  as  the  total  team  hours 
for  each  assessment.  The  results  of  this  computation  are  shown  below.  42%  (35  out  of  83)  of 
the  assessments  spent  between  1 1  to  20%  of  their  total  team  hours  on  consolidation  activities. 
The  average  proportion  is  20%. 


Proportion  of  onsite  team  hours  spent  on 
consolidation  activities 
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Figure  29:  Proportion  of  On-Site  Team  Hours  Spent  on  Consolidation  Activities 


5.2.3  Team  Hours  Required  to  Perform  CBA IPI  Activities 

Figure  30  below  shows  the  time  periods  indicated  to  perform  the  pre-onsite  and  the  onsite 
activities,  as  well  as  time  spent  in  consolidation  activities.  The  chart  shows  the  largest  ob¬ 
served  value,  the  smallest  observed  value,  the  median  values,  and  the  25th  and  75th  percentiles 
for  each  of  the  three  sets  of  measures. 
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Figure  30:  Time  to  Perform  CBA  IPI  Activities 
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6  Conclusion 


6.1  CBA IPI  Requirements 

Overall,  the  data  from  the  Lead  Assessor  Requirements  Checklists  submitted  for  83  assess¬ 
ments  indicate  that  there  are  no  significant  problems  in  meeting  the  CBA  IPI  requirements. 

6.2  Key  Findings 

The  results  of  analyzing  the  data  submitted  through  the  Lead  Assessor  Requirements  Check¬ 
list  have  produced  some  useful  information  that  may  provide  good  references  for  future  as¬ 
sessments.  The  following  table  summarizes  some  of  the  key  findings  in  this  document  that 
may  be  useful  references  for  Lead  Assessors: 


Item 

Findings 

Planning  the  Assessment 

Team  size 

Range:  4  to  13  team  members 

Average:  7  team  members 

Business  goals 

Top  three: 

1 .  “Faster,  better,  cheaper” 

2.  Attain  Level  2  Maturity 

3.  Identify  improvement  areas 

Organization  size 

Range:  5  to  934  people  involved  in  software  development 

CMM  training 

31%  SEI  and  64%  non-SEI 

CBA  IPI  training 

Most  frequent  delivery  time:  16-20  hours 

Supplementary  materials 

Top  3  areas  supplemented: 

1 .  Automation  tools 

2.  Exercises 

3.  Planning  &  team  building 

Conducting  the  Assessment 

Maturity  questionnaires 

Range:  0  to  47  questionnaires 

Most  frequent  number  administered:  0  to  5  questionnaires 

Interviews 

Most  frequent  total  number  of  interviews:  21  to  30 

Documents  reviewed 

Most  frequent  number  of  documents  reviewed:  10  to  100 

Observations 

Range:  7  to  3000  observations 

Draft  findings 

Range  of  draft  findings  presented:  1  to  236 

Most  frequent  number  of  presentations:  2 

Additional  Questions 

Length  of  assessments 

Most  frequent  length:  5  days 

Average:  9  days 

Team  hours 

Median  number  of  team-hours  for  pre-onsite  activities:  35 
Median  number  of  team-hours  for  on-site  activities:  62 
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Item 

Findings 

Median  number  of  team-hours  for  consolidation:  19 

Median  number  of  total  team-hours:  97 

6.3  Lead  Assessor  Requirements  Checklist 

Analysis  of  the  data  included  in  this  report  has  provided  considerable  insight  into  the  meas¬ 
ures  related  to  conducting  a  CBA IPI  assessment.  It  has  also  provided  guidance  that  will  help 
towards  improving  the  checklist  when  it  is  incorporated  as  one  of  several  feedback  forms  to 
reside  on  the  Lead  Assessor  Web  Center. 

Many  thanks  to  each  of  the  Lead  Assessors  who  provided  this  data.  We  hope  that  the  check¬ 
list  was  a  useful  planning  tool  for  you  and  not  just  added  administrative  overhead.  Your  con¬ 
tributions  of  data  provide  a  tool  for  yourself  and  other  Lead  Assessors  in  doing  a  better  job  of 
planning  a  CBA  IPI  for  future  assessment  opportunities. 
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